Method and Apparatus for Communication Between Humans and Devices

ABSTRACT

This invention relates to methods and apparatus for improving communications between humans and devices. The invention provides a method of modulating operation of a device, comprising: providing an attentive user interface for obtaining information about an attentive state of a user; and modulating operation of a device on the basis of the obtained information, wherein the operation that is modulated is initiated by the device. Preferably, the information about the user&#39;s attentive state is eye contact of the user with the device that is sensed by the attentive user interface.

FIELD OF THE INVENTION

This invention relates to attentive user interfaces for improvingcommunication between humans and devices. More particularly, thisinvention relates to use of eye contact/gaze direction information bytechnological devices and appliances to more effectively communicatewith users, in device or subject initiated communications.

BACKGROUND OF THE INVENTION

Interaction with technological devices is becoming an ever-increasingpart of everyday life. However, effectiveness and efficiency of suchinteraction is generally lacking. In particular, when seeking userinput, devices such as computers, cellular telephones and personaldigital assistants (PDAs) are often disruptive, because such devicescannot assess the user's current interest or focus of attention. Moreefficient, user-friendly interaction is desirable in interactions withhousehold appliances and electronic equipment, computers, and digitaldevices.

One way that human-device interactions can be improved is by employinguser input such as voice and/or eye contact, movement, or position toallow users to control the device. Many previous attempts relate tocontrolling computer functions by tracking eye gaze direction. Forexample, U.S. Pat. Nos. 6,152,563 to Hutchinson et al. and 6,204,828 toAmir et al. teach systems for controlling a cursor on a computer screenbased on user eye gaze direction. U.S. Pat. Nos. 4,836,670 and 4,973,49to Hutchinson, U.S. Pat. No. 4,595,990 to Garwin et al., U.S. Pat. No.6,437,758 to Nielsen et al., and U.S. Pat. No. 6,421,064 and U.S. PatentApplication No. 2002/0105482 to Lemelson et al. relate to controllinginformation transfer, downloading, and scrolling on a computer based onthe direction of a user's eye gaze relative to portions of the computerscreen. U.S. Pat. No. 6,456,262 to Bell provides an electronic devicewith a microdisplay in which a displayed image may be selected by gazingupon it. U.S. Patent Application No. 2002/0141614 to Lin teachesenhancing the perceived video quality of the portion of a computerdisplay corresponding to a user's gaze.

Use of eye and/or voice information for interaction with devices otherthan computers is less common. U.S. Pat. No. 6,282,553 teachesactivation of a keypad for a security system, also using an eye tracker.Other systems employ detection of direct eye contact. For example, U.S.Pat. No. 4,169,663 to Murr describes an eye attention monitor whichprovides information simply relating to whether or not a user is lookingat a target area, and U.S. Pat. No. 6,397,137 to Alpert et al. relatesto a system for selecting left or right side-view mirrors of a vehiclefor adjustment based on which mirror the operator is viewing. U.S. Pat.No. 6,393,136 to Amir et al. teaches an eye contact sensor fordetermining whether a user is looking at a target area, and using thedetermination of eye contact to control a device. The Amir et al. patentsuggests that eye contact information can be used together with voiceinformation, to disambiguate voice commands when more than onevoice-activated devices are present.

While it is evident that considerable effort has been directed toimproving user-initiated communications, little work has been done toimprove device-initiated interactions or communications.

SUMMARY OF THE INVENTION

According to a first aspect of the invention there is provided a methodof modulating operation of a device, comprising: providing an attentiveuser interface for obtaining information about an attentive state of auser; and modulating operation of a device on the basis of said obtainedinformation, wherein said operation that is modulated is initiated bysaid device.

In a preferred embodiment, said information about said user's attentivestate is eye contact of said user with said device that is sensed bysaid attentive user interface. In another embodiment, said informationabout said user's attentive state is eye contact of said user with asubject that is sensed by said attentive user interface. In oneembodiment, said subject is human, and said information about saiduser's attentive state is eye contact of said user with said human thatis sensed by said attentive user interface. In another embodiment, saidsubject is another device. In accordance with this embodiment, when saiduser's attention is directed toward said other device, said modulatingstep comprises routing a notification to said other device. In variousembodiments, said information about an attentive state of said user isbased on one or more indices selected from the group consisting of eyecontact, eye movement, eye position, eye gaze direction, voice, bodypresence, body orientation, head and/or face orientation, user activity,and brain activity/arousal.

In one embodiment of the method said sensing of eye contact comprises:obtaining successive full-frame video fields of alternating bright anddark video images of said user's pupils; and subtracting said imagesbetween frames to locate said pupils; wherein locating said pupilsconfirms eye contact of said user. In a preferred embodiment, saidsensing of eye contact further comprises: detecting a glint in theuser's eyes; and confirming eye contact of said user when said glint isaligned with said pupils.

In accordance with the first aspect of the invention, when said user'sattention is not directed toward said device, said modulating stepcomprises notifying said user progressively, from a less interruptivenotification to a more interruptive notification. In variousembodiments, said notification is of at least one type selected from thegroup consisting of audio, visual, and tactile.

In various embodiments, said attentive user interface may be attached toor embedded in said device, or attached to or embedded in a member ofthe group consisting of clothing, eyewear, jewelry, and furniture. Insome embodiments, the device may be a personal computer, a cellulartelephone, a telephone, a personal digital assistant (PDA), or anappliance.

In various embodiments, said modulating step may comprise forwardingsaid obtained information to another device or a network of devices,modulating a notification being sent to said user, or forwarding saidobtained information to another device or a network of devices.

According to a second aspect of the invention there is provided a methodof modulating operation of a network of devices, comprising: providingeach device of a network of devices with an attentive user interface forobtaining information about an attentive state of a user with respect toeach device; and modulating operation of said devices on the basis ofsaid obtained information, wherein said operation that is modulated isinitiated by at least one of said devices.

In various embodiments, said operation that is modulated may comprisenotification, communication, information transfer, and a combinationthereof, or routing said notification, communication, informationtransfer, or combination thereof, to a device with which said user isengaged. The modulating operation may further comprise modulatingnotification of said user progressively, from a less interruptivenotification to a more interruptive notification. In a preferredembodiment, said information about said user's attentive state is eyecontact of said user with each said device, said eye contact beingsensed by said attentive user interface.

According to a third aspect of the invention there is provided a methodof modulating communication over a network of at least two devices,comprising: providing a first device of a network of devices with anattentive user interface for obtaining information about a first user'sattentive state toward said first device; providing a second device of anetwork of devices with an attentive user interface for obtaininginformation about a second user's attentive state toward said seconddevice; providing said first device of said network with a proxy forcommunicating to said first user said information about said seconduser's attentive state toward said second device; providing said seconddevice of said network with a proxy for communicating to said seconduser said information about said first user's attentive state towardsaid first device; relaying to said network said information about saidfirst and second users' attentive states toward said respective firstand second devices; wherein communication between said first and seconddevices is modulated on the basis of the attentive states of said firstand second users toward their respective devices.

In one embodiment, communication between said first and second devicesis enabled when respective proxies indicate that attentive states ofsaid first and second users are toward respective devices. In otherembodiments, the device may be a telephone, and the proxy may be arepresentation of a user's eyes. In a further embodiment, the networkcomprises more than two devices.

According to a fourth aspect of the invention there is provided a methodof modulating operation of a cellular telephone, comprising: providingan attentive user interface for obtaining information about an attentivestate of a user; and modulating operation of a cellular telephone on thebasis of said obtained information, wherein said operation that ismodulated is initiated by said cellular telephone. In a preferredembodiment, said information about said user's attentive state is eyecontact of said user with said cellular telephone that is sensed by saidattentive user interface.

According to a fifth aspect of the invention there is provided a methodof modulating operation of a graphical user interface, comprising:providing a graphical user interface for displaying one or more imagesto a user; determining said user's eye gaze direction to obtaininformation about which image is being viewed by said user; and usingsaid information to enlarge, on said graphical user interface, saidimage being viewed by said user, and to shrink, on said graphical userinterface, one or more images not being viewed by said user, whereinsaid enlarging of an image does not obscure said one or more images notbeing viewed.

According to a sixth aspect of the invention there is provided anapparatus for detecting eye contact of a subject looking at a user,comprising an eye contact sensor worn by said user that indicates eyecontact of a subject looking at the user. In a preferred embodiment, theapparatus comprises eyeglasses.

According to a seventh aspect of the invention there is provided an eyecontact sensor, comprising: an image sensor for obtaining successivefull-frame video fields of alternating bright and dark video images of auser's pupils; and means for subtracting said images between frames tolocate said pupils; wherein said located pupils indicate eye contact ofsaid user. In a preferred embodiment, the eye contact sensor furthercomprises means for detecting alignment of a glint in said user's eyeswith said user's pupils; wherein alignment of said glint with saidpupils indicates eye contact of said user.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described below, by way of example,with reference to the accompanying drawings, wherein:

FIG. 1 is a schematic diagram of an eye contact sensor;

FIG. 2 depicts an algorithm for an eye contact sensor in accordance withan embodiment of the invention;

FIG. 3 depicts an algorithm for an attentive user interface inaccordance with an embodiment of the invention;

FIG. 4 shows eye glasses equipped with an eye contact sensor inaccordance with an embodiment of the invention;

FIG. 5 is a schematic diagram of a device equipped with a mechanical eyeproxy and an eye contact sensor in accordance with an embodiment of theinvention; and

FIG. 6 depicts a scheme for telephone eye proxy in accordance with anembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is based, at least in part, on the recognitionthat human-device interaction can be improved by implementing in devicessome of the basic social rules that govern human face-to-faceconversation. Such social rules are exemplified in the followingscenario: Person A is in conversation with person B (or engaged in atask), and person C wishes to gain A's attention. There are a number ofways in which C may do so without interfering with A's activities.Firstly, C may position himself such that A becomes peripherally awareof his presence. Secondly, C may use proximity, movement, gaze or touchto capture A's attention without using verbal interruption. The use ofnonverbal visual cues by C allows A to finish his conversation/taskbefore acknowledging C's request for attention, e.g., by making eyecontact. If A does not provide acknowledgement, C may choose to withdrawhis request by moving out of A's visual field. Indeed, Frolich (1994)found that initiators of conversations often wait for visual cues ofattention, in particular, the establishment of eye contact, beforelaunching into their conversation during unplanned face-to-faceencounters. Face-to-face interaction is therefore different from the waywe typically interact with most technological devices in that itprovides a rich selection of both verbal and nonverbal communicationchannels. This richness is characterized by (i) flexibility in choosingalternate channels of communication to avoid interference orinterruption, (ii) a continuous nature of the information conveyed, and(iii) a bi-directionality of communication.

Electronic devices that require user input or attention do not followsuch social rules in communicating with users. As a result they oftengenerate intrusive and annoying interruptions. With the advent ofdevices such as cell phones and personal digital assistants (PDAs; e.g.,Blackberry®, Palm Pilot®), users are regularly interrupted with requestsfor their attention. The present invention solves this problem byaugmenting devices with attentive user interfaces: user interfaces thatnegotiate the attention they receive from or provide to users bynegotiations through peripheral channels of interaction. Attentive userinterfaces according to the invention follow social rules of human groupcommunication, where, likewise, many people might simultaneously have aninterest in speaking. In human group conversations, eye contactfunctions as a nonverbal visual signal that peripherally conveys who isattending to whom without interrupting the verbal auditory channel. Withit, humans achieve a remarkably efficient process of conversationalturn-taking. Without it, turn-taking breaks down. Thus, an attentiveuser interface according to the invention applies such social rules todevice-initiated interactions or communications, by assessing a user'sattentive state, and making a determination as to whether, when, and howto interrupt (e.g., notify) the user on the basis of the user'sattentive state.

To facilitate turn-taking between devices and users in a non-intrusivemanner, an attentive user interface according to the invention assessesa user's attentive state by sensing one or more parameters of the user.Such parameters are indicative of the user's attentive state, andinclude, but are not limited to, eye contact, eye movement, eyeposition, eye gaze direction, voice, body presence, body orientation,head and/or face orientation, activity, and brain activity/arousal. Inthe case of eye contact, movement, or position, an attentive userinterface senses the eyes of the user, or between the user and a subject(e.g., another human), to determine when, whether, and how to interruptthe user. For example, notification by a PDA seeking user input can bemodulated on the basis of whether the user is engaged with the PDA, withanother device, or a subject. The PDA then can decide whether, when, andhow to notify; for example, directly, or indirectly via another devicewith which the user is engaged. Body presence can be sensed in variousways, such as, for example, a motion detector, a radio frequency (RF) IDtag worn by a user and sensed using, e.g., BlueTooth®, a visual tag,electro-magnetic sensors for sensing presence/location/orientation of auser within a magnetic field, and a global positioning system (GPS).

As used herein, the term “user” is intended to mean the entity,preferably human, who is using a device.

As used herein, the term “device” is intended to mean any digitaldevice, object, machine, or appliance that requires, solicits, receives,or competes for a user's attention. The term “device” includes anydevice that typically is not interactive, but could be made moreuser-friendly by providing interaction with a user as described herein.

As used herein, the term “subject” is intended to mean the human,device, or other object with which a user might be engaged.

As used herein, the term “attentive user interface” is intended to meanany hardware and/or software that senses, receives, obtains, andnegotiates a user's attention by sensing one or more indices of a user'sattentive state (e.g., eye contact, eye movement, eye position, eye gazedirection, voice, body presence, body orientation, head and/or faceorientation, activity, brain activity/arousal), with appropriatehardware and associated algorithms and/or software for interfacing theattentive user interface with a device or a network of devices. Anattentive user interface comprises portions for sensing user attentivestate and for processing and interfacing/relaying information about theuser's attentive state to a device. Such portions can be housed as aunit or as multiple units. Interfacing an attentive user interface witha device comprises providing an output from the attentive user interfaceto the device, which controls operation of the device. An attentive userinterface of the invention can perform one or more tasks, such as, butnot limited to, making decisions about user presence/absence, makingdecisions about the state of user attention, prioritizing communicationsin relation to current priorities in user attention as sensed by theattentive user interface, modulating channels and modes of delivery ofnotifications and/or information and/or communications to the user,modulating presentation of visual or auditory information, andcommunicating information (e.g., indices) about user attention to othersubjects.

As used herein, the term “attentive state” is intended to mean a measureor index of a user's engagement with or attention toward a subject.Examples of such indices are eye contact, eye movement, eye position,eye gaze direction, voice, body presence, body orientation, head and/orface orientation, activity, and brain activity/arousal.

As used herein, the term “notify” or “notification” is intended to meanthe signalling or soliciting, usually by a device, for a user'sattention. For example, notification can employ any cue(s) that act on auser's senses to solicit the user's attention, such as one or more ofaudio, visual, tactile, and olfactory cues.

As used herein, the term “modulating” is intended to mean controlling,enabling and/or disabling, or adjusting (e.g., increasing and/ordecreasing). With respect to notification, modulating includes, forexample, turning notification on or off, delaying notification, changingthe volume or type of notification, and the like. For example,notification can be gradually modulated from less interruptive (e.g.,quiet) to more interruptive (e.g., loud), as time passes without useracknowledgement. Modulating also refers to changing the vehicle orchannel for notification, communication, or data transfer; for example,by routing such through a network to a more appropriate device. Forexample, in the case of an urgent notification, modulation mightencompass routing the notification to a device with which the user isengaged, increasing the likelihood that the user receives thenotification (see Example 4, below).

As used herein, the terms “mediated communication” and “mediatedconversation” refer to communication or conversation that takes placethrough a medium such as video or audio devices/systems, such that thereis no face-to-face conversation between the participants. In mostmediated communications, participants involved are remotely locatedrelative to one another.

In one embodiment of the invention, an attentive user interfacedynamically prioritizes the information it presents, and the way it ispresented, to a user, such that information processing resources of bothuser and system are optimally used. This might involve, for example,optimally distributing resources across a set of tasks. An attentiveuser interface does this on the basis of knowledge—consisting of acombination of measures and models—of the present, and preferably alsothe past and/or future states of the user's attention, taking intoaccount the availability of system resources. Attentive user interfacesmay employ one or more of eye contact, eye movement, eye position, eyegaze direction, voice, body presence, body orientation, head and/or faceorientation, activity, brain activity/arousal to detect attentive state.Attentive user interfaces may store any of the above measures as amodel, used to govern decisions about the user's attentive state.

In a preferred embodiment, an attentive user interface employs eyecontact and/or eye gaze direction information, optionally in combinationwith any further measures of user presence mentioned above. Eye contactsensors as used in the invention are distinguished from eye trackers, inthat eye contact sensors detect eye contact when a subject or user islooking at the sensor, whereas eye trackers detect eye movement todetermine the direction a subject or user is looking.

In some embodiments, an attentive user interface employs an eye contactsensor based on bright-dark pupil detection using a video camera (see,for example, U.S. Pat. No. 6,393,136 to Amir et al.). This techniqueuses intermittent on-camera axis and off-camera axis illumination of theeyes to obtain an isolated camera image of the user's pupil. The on-axisillumination during one video field results in a clear reflection of theretina through the pupil (i.e., the bright pupil effect). Thisreflection does not occur when the eyes are illuminated by the off-axislight source in the next video field. By alternating on-axis withoff-axis illumination, synchronized with the camera clock, successivevideo fields produce alternating bright and dark images of the pupil. Bysubtracting these images in real time, pupils can easily be identifiedwithin the field of view of a low-cost camera. Preferably, eyes areilluminated with infrared (IR) light, which does not distract the user.

However, accuracy of the eye contact sensor can be improved by measuringthe glint, or first purkinje image, of the eyes. The glint is areflection of light on the outer side of the cornea, that acts as arelative reference point, which can be used to eliminate the confoundingeffects of head movements. The glint moves with the head, but does notrotate with the pupil because the eye is spherical. Thus, the positionof the glint relative to the pupil can be used to determine thedirection a user or subject is looking. For example, when the user islooking at the camera and the glint is inside the pupil, the pupil,glint, and camera are aligned on the camera axis, indicating that theuser is looking at the camera, and hence eye contact is detected.

We have used this technique in attentive user interfaces to identify eyecontact of users at approximately 3 meters distance, using standard320×240 CCD cameras with analog NTSC imaging. The ability to obtain areliable estimate of the pupils at larger distances is limited by theresolution of such cameras. Use of mega-pixel CCD cameras, althoughexpensive, make possible the detection of pupils at greater distances.Alternatively, high-resolution CMOS imaging technology (e.g., SiliconImaging MegaPixel Camera SI-3170U or SI-3200U) allows the manufacture oflow-cost high-resolution eye contact sensors.

An example of a high-resolution eye contact sensor is shown in FIG. 1.The high-resolution eye contact sensor 40 comprises an image sensor(i.e., a camera), such as a black and white high-resolution CCD or CMOSimage sensor (3 Mpixels or more), with a multifocus lens 48. Preferably,infrared light is used to illuminate the eyes, and accordingly aninfrared filter is disposed beneath the lens 48. The output of the imagesensor is connected to circuitry which uses the camera frame sync signalto illuminate the space in front of the camera with on-axis lightproduced by, e.g., an array of infrared LEDs 42, and off-axis lightproduced by, e.g., two arrays of infrared LEDs 44,52. On-axis andoff-axis light is produced alternately with odd and even frames. Forexample, on-axis light is produced each odd frame and off-axis light isproduced every even frame. Images are processed to locate theuser's/subject's eyes, and corresponding information is relayed tohardware/software of an attentive user interface. The information isused by the attentive user interface to determine, whether, how, when,etc., to interrupt or send a notification to a user. In some embodimentsthe image processing circuitry and software may reside in the eyecontact sensor unit 40, whereas in other embodiments the circuitry andsoftware are remote (e.g., associated with a host computer) and suitablyconnected to the eye contact sensor unit 40 using, e.g., ahigh-bandwidth video link, which can be wireless, such as Apple®FireWire® or USB 2 based. As shown in the eye protocol specificationbelow, information relating to eye contact may include whether eyes arefound in the image, where the eyes are, how many eyes are present,whether the eyes are blinking, and if the unit is calibrated, what theeyes are looking at in screen coordinates. The information may alsoinclude a flag for each eye when the eyes are looking straight at thecamera.

Eye Protocol Specification { }=Data set ( )=Subset 1. EYE_NOT_FOUND

ID End 0 CR & LF ASCII CR = 77 or 4D

2. HEAD_FOUND

ID D1 D2 End 1 Number of Heads {(T L B R)₁ . . . (T L B R)₉} CR & LF D1: Number of Heads D1 = {1, . . . , 9 } D2 : Head Boundary Box D2 = {(TopLeft Bottom Right)_(1,) . . . , (Top Left Bottom Right)₉} Numbers inASCII format (unsigned int) separated by ASCII space

3. EYE_FOUND

ID D1 D2 End 2 Number of Eyes {(X_(g)Y_(g)X_(p)Y_(p))₁ . . .(X_(g)Y_(g)X_(p)Y_(p))₉} CR & LF D1 : Number of Eyes D1 = {1, . . . , 9} D2 : Glint and pupil Coordinate D2 = ((X_(g) Y_(g) X_(p) Y_(p))_(1,) .. . , (X_(g) Y_(g) X_(p) Y_(p))₉) Numbers in ASCII format (unsigned int)separated by ASCII space

4. EYE_BLINK

ID D1 D2 End 3 Number of Eyes {F₁ . . . F₉ } CR & LF D1 : Number of EyesD1 = {1, . . . , 9 } D2 : Blink D2 F = {0 , 1} 0 = NOT_BLINK 1 = BLINKNumbers in ASCII format (unsigned int) separated by ASCII space

5. EYE_CONTACT

ID D1 D2 End 4 Number of Eyes {F₁ . . . F₉} CR & LF D1 : Number of EyesD1 = {1 , . . . , 9} D2 : Eye Contact D2 F = {0, 5} 0 = No Contact 5 =Contact Numbers in ASCII format (unsigned int) separated by ASCII space

6. CALIBRATED_SCREEN_COORDINATE

ID D1 End 5 (x, y) CR & LF D1 : Screen Coordinate (x, y) Numbers inASCII format (unsigned int) separated by ASCII space

Preferably, the eye contact sensor determines the orientation of pupilswith a spatial accuracy of, for example, 1 meter at 5 meters distance(about 10 degrees of arc) and a head movement tolerance of, for example,20 degrees of arc, at a distance of 5 meters or more. For bestperformance, the frame rate of the eye contact sensor's camera should beas high as possible, and in the order of 100 Hz. The effective samplingrate of the sensor preferably corresponds to at least 20 Hz, given thatthe minimum human fixation time is in the order of 100 ms.

It should be noted that the use of a subtraction algorithm to locatepupils results in a tradeoff between temporal and spatial resolution. Inone embodiment, in which image subtraction occurs within frames (see,e.g., U.S. Pat. No. 6,393,136 to Amir et al.), resulting in an effectivespatial resolution of the sensor of only half that of the camera. Here,the image processing algorithm and LEDs are synchronized with half-framefields generated by an NTSC or other interlaced camera technology.

However, the invention provides, in one aspect, a method and apparatusfor obtaining eye contact information in which image subtraction occursbetween frames (by subtracting an odd frame from an even frame, or viceversa), as shown in the algorithm of FIG. 2. This allows the use of thefull camera resolution, and thus a greater tracking range, whilereducing the effective frame or sampling rate by half. The subtractionalgorithm and LEDs are synchronized with a full frame clock generated bythe camera and the minimum sampling frequency of the camera ispreferably in the order of about 30 to about 40 Hz.

In other embodiments an attentive user interface uses eye gaze directionas input about a user's attentive state. Eye gaze direction is detectedby an eye tracker, such as that described in detail in U.S. Pat. No.6,152,563 to Hutchinson et al.

An attentive user interface of the invention may be applied touser-initiated control of a device using, for example, eye contactand/or eye gaze direction, with or without further input, such as voice,body presence, and the like. However, the invention is particularlyapplicable to device-initiated communication with a user, such as, forexample, notifying a user of an incoming message, or of a task requiringuser input. As shown in FIG. 3, an attentive user interface, running ona such a device, senses and evaluates one or more indices of userattention (e.g., eye contact, eye movement, eye position, eye gazedirection, voice, body presence, body orientation, head and/or faceorientation, activity, brain activity/arousal) to determine whether,when, and how to notify, interrupt, respond or respond to the user,open/close communication channels, an the like. By progressivelysampling the user's attention, and appropriately signalingnotifications, the user can be notified with minimal interruption. Forexample, as shown in FIG. 3, an attentive user interface mightprogressively signal for the user's attention. Initially this may happenthrough a channel that is peripheral to the user's current activity. Theinterface may then wait for user acknowledgement, provided through,e.g., an input device, before opening a direct channel to the user. If,however, no user acknowledgement is received within a given period, theattentive user interface may proceed to a more direct channel to theuser, increase the urgency level of the notification, or defernotification.

In one embodiment, information obtained about a user's attentive stateis communicated to one or more subjects who might wish to contact theuser. Such communication can be through any network by which the userand subject(s) are connected, such as a local area network, a wide areanetwork (e.g., the internet), or hard-wired or wireless (e.g., cellular)telephone network. Subjects can evaluate the information about theuser's attentive state, and, using rules of social engagement, decidewhether or not to contact the user. For example, in telephoniccommunications (as described in detail in Example 1), information aboutthe user's current attentive state is communicated to a subjectattempting to telephone the user. The subject can decide whether toproceed with the telephone call on the basis of such information.

Further, the invention provides for an environment in which multipledevices, each equipped with attentive user interfaces, are networked,such that information concerning to which device the user's attention isdirected is available to all devices on the network. By progressivelysignaling notifications (e.g., in the case of a cell phone, the phonestarts by ringing quietly and progressively rings louder depending onurgency of the call and/or proximity to the user; or, an icon on thecell phone's screen changes as urgency increases), and by determiningwhich device the user is currently attending to, a notification and/ormessage can be forwarded to the appropriate device so that the messageis received with minimal interruption of the user's primary task.

There are numerous applications of an attentive user interface accordingto the invention, in addition to those discussed above. In someembodiments, the hardware component of the attentive user interface issmall and light weight, such that it can be embedded in or attached to apersonal electronic device such as a cell phone, jewelry, clothing, oreyeglasses, and the like. For example, FIG. 4 shows a front view of apair of eye glasses having an eye contact sensor attached thereto. Theeye glasses 2 have a frame 4 and lenses 6 and 8. A camera lens 10 isembedded in the frame 4 of the glasses, pointing outward. Surroundingthe camera lens 10 is an array of on-axis LED illuminators 12. Two rowsof off-axis LED illuminators 14,16 are positioned near the outerperipheries of the lenses 6,8. The camera feed as well as the LED arraysare connected through wires to a control unit worn by the user. Thiscontrol unit contains power and circuitry for illumination of the LEDsand camera sychronization. In one embodiment, the control unit performscomputer vision processing according to an algorithm using an embeddedprocessor board. In such an embodiment, data is sent over a wireless orwired network link to a host. In another embodiment, camera images aresent over a wireless or wired network to an external computer visionprocessing facility. Eye contact glasses can be used, for example, toopen/close communication channels between co-located but distant users,or for regulating messaging to a user or between two or more users.

One application of eye contact glasses is to track how many individualshave looked at the user during a specified period. These data orstatistics can be made available on the user through an LCD display, orsent to a networking device for further processing or display. Combinedwith computer vision or other means, the eye contact glasses candetermine who has looked at the user, for how long, and when. In oneembodiment, the eye contact glasses provides a personal attention sensor(i.e., a “hit counter”), which indicates to a user when he/she is beinglooked at by a subject. For example, a counter could be incrementedwhenever the user has been looked at by a subject, to provideinformation about the number of “hits”. Such an embodiment can provideamusement to users in certain social settings.

In other embodiments, an attentive user interface of the inventionincludes a sensor for detecting one or more indices of user attentivestate in combination with a “proxy”.

As used herein, the term “proxy” is intended to mean any hardware orvirtual (e.g., an image on a computer screen) representation of a(remote) subject's attention. For example, a proxy can be a pair ofeyes, either mechanical or virtual (e.g., pictured on a computerscreen), that inform a user of the state of attention of a subject withwhich the user is attempting to establish mediated communication (e.g.,via telephone). Eye proxies are preferred because of what theyrepresent; that is, the establishment of eye contact is related to theestablishment of communication between individuals.

In such embodiment, an attentive user interface, including a proxy, isused not only to obtain information about the attention of its user, butalso functions to communicate robot, machine, or remote user attentiondirected towards a user. For example, an eye contact sensor can bemounted on a robotic actuation device that allows rotation of the eyecontact sensor in 3 orientation directions. The eye contact sensorfunctions as virtual eyes directing the robotic device in establishingeye contact with the user when the attentive user interface's attentionis directed towards that user. To convey attention, the robotic devicemay feature a pair of mechanical eyes, or an image or video of a remoteuser or computer agent. FIG. 5 shows an embodiment in which a pair ofrobotic mechanical eyes 60 and an eye contact sensor with camera lens62, on-axis LED array 64, and off-axis LED arrays 66,68 are mounted on adevice 70, such as a telephone.

In accordance with this embodiment, an attentive user interface with asensor such as an eye contact sensor or an eye tracker can be used withany device to sense whether a user is available for communication, andwhether a user is communicating with that device, via any route such asa keyboard, speech recognition, or manual interactions. Conversely, aproxy can signal the device's attention to the user by alignment of theeye contact sensor and/or virtual eyes with the user's eyes. If thedevice has not recently received visual attention from the user, itchooses an unobtrusive method to signal the user (i.e., by vibrating,rotating its eyeballs to obtain attention or any other nonverbal means).A device remains in the periphery of user activity until the user hasacknowledged the device's request for attention. At that time that thedevice receives user attention, as measured with the eye contact sensoror through other means, a mediated communication channel with the useris established, including, for example, speech production or display ofinformation. Example 2 describes an example of this embodiment indetail.

In further embodiments, an attentive user interface can be embedded indigital devices such as computers, personal digital assistants (PDAs),pvr/tv/vcr/cameras, telephones, household appliances, furniture,vehicles, and any other location where information about a user'sattentive state can advantageously be used to modulate their behavior(see the Examples, below). An attentive user interface can be used tocontrol video and audio recording and transmission, or to senseattention during remote or colocated meeting for retroactive automatedediting (i.e., a virtual director), or for video conferencing cameraselection and remote personal attention sensing (see Example 3, below).Yet other applications include, but are not limited to, remote (instant)messaging (i.e., open/close communication with a user at a distance,such as during remote arbitrage); colocated messaging (i.e., open/closecommunication with a user at a physical distance); dynamic email filterbased on time spent reading; intelligent agent communication ofattention; robot communication of attention; avatar/remote personcommunication of attention; presence detection for any kind of messagingsystem; receipt of message acknowledgement for any kind of system;notification negotiation (i.e., user acknowledgement of informationpresentation); notification optimization (i.e., forwarding to currentdevice); optimization of information presentation (i.e., presentnotification or other information on device or part of device where useris looking); for pointing to items on displays; to determine target ofkeyboard commands; look to talk; eye telepointing systems (i.e.,presentation and remote collaboration); vehicle navigation systemoperation (selection of information retrieval system); vehicle phonecall answering; vehicle operator fatigue sensor; visualization andmonitoring of user attention (see Example 4); attentive reasoningnetworks for telecommunication for telemarketeering purposes (e.g.,determine where users are and what they pay attention to (see Example5), to forward calls, or to data-mine subjects in user's attention);displaying networks of attention between users or between users andsubjects; surveillance and security camera monitoring; and modifying thesize, resolution, or content of a window on a graphical user interface(see Examples 6 and 7).

The contents of all cited patents, patent applications, and publicationsare incorporated herein by reference in their entirety.

The invention is further described by way of the following non-limitingexamples.

Example 1 Attentive Cell Phone

In this example, an attentive user interface was used to apply some ofthe basic social rules that surround human face-to-face conversation(discussed above) to a personal electronic device, in this case a cellphone. However, the embodiment described in this example could beimplemented in any electronic device or appliance.

The subtlety of interruption patterns typically used during humanface-to-face communication is completely lost when using cell phones.Firstly, a person making a call usually is unaware of the status ofinterruptability of the user being called. Secondly, there is limitedfreedom in choosing alternative channels of interruption. Thirdly, thechannels that do exist do not allow for any subtlety of expression. Inthis example, an attentive cell phone was created by augmenting a CompaqiPAQ handheld with an attentive user interface employing a low-costwearable eye contact sensor for detecting when a user is in aface-to-face conversation with another human.

Wearable microphone headsets are becoming increasingly common with cellphones. The signal from such microphones is available with high fidelityeven when the user is not making a call. We modified the cell phone toaccept such input, allowing it to monitor user speech activity toestimate the chance that its user is engaged in a face-to-faceconversation. Wireless phone functionality was provided by voice-over-ipsoftware connected through a wireless LAN to a desktop-based callrouter. An attentive state processor running on the same machine sampledthe energy level of the voice signal coming from the cell phone. Toavoid triggering by non-speech behavior we used a simplified version ofa turn detection algorithm described by Vertegaal (1999). That is, whenmore than half the samples inside a one-second window indicate speechenergy, and those samples are evenly balanced across the window, theprobability of speech activity by its user is estimated at 100%. Foreach second that the user is silent, 5% is subtracted from thisestimate, until zero probability is reached. Thus we achieved ashort-term memory of 20 seconds for speech activity by its user.

Speech detection works well in situations where the user is the activespeaker in conversation. However, when the user is engaged in prolongedlistening, speech detection alone does not suffice. Given that there isno easy way to access the speech activity of an interlocutor withoutviolating privacy laws, we used an alternative source of input, eyecontact.

According to Vertegaal (1999), eye tracking provides an extremelyreliable source of information about the conversational attention ofusers. In dyadic conversations, speakers look at the eyes of theirconversational partner for about 40% of the time. The eye contact sensordetected eye gaze toward a user by an interlocutor (i.e., a subject) todetermine when the user was engaged in a conversation with the subject.In one embodiment, the contact sensor was worn on a cap worn on theuser's head. In another embodiment, the sensor was embedded in the eyeglasses worn by the user (see above and FIG. 4). The sensor consisted ofa video camera with a set of infrared LEDs mounted on-axis with thecamera lens. Another set of LEDs was mounted off-axis.

By synchronizing the LEDs with the camera clock, bright and dark pupileffects were produced in alternate fields of each video frame. A simplealgorithm found any eyes in front of the user by subtracting the evenand odd fields of each video frame (Morimoto, 2000). The LEDs alsoproduced a reflection from the cornea of the eyes. These glints appearednear the center of the detected pupils when the subject was looking atthe user, allowing the sensor to detect eye contact without calibration.By mounting the sensor on the head, pointing outwards, the sensor'sfield of view was always aligned with that of the user. Sensor data wassent over a TCP/IP connection to the attentive state processor, whichprocesses the data using an algorithm similar to that used for speech todetermine the probability that the user received gaze by an onlooker inthe past 20 seconds.

The attentive state processor determined the probability that the userwas in a conversation by summating the speech activity and eye contactestimates. The resulting probability was applied in two ways. Firstly,it set the default notification level of the user's cell phone.Secondly, it was communicated over the network to provide informationabout the status of the user to potential callers.

Communicating Attentive State to Callers

When the user opens his/her contact list to make a phone call, theattentive phone updates the attentive state information for all visiblecontacts. In this example, below the contact's name a menu shows thepreferred notification channel. Notification channels are listedaccording to their interruption level: message; vibrate; private knock;public knock; and public ring. Users can set their preferred level ofinterruption for any attentive state. They can also choose whether toallow callers to override this choice. When contacts are available forcommunication, their portraits display eye contact. A typical preferrednotification channel in this mode is a knocking sound presentedprivately through the contact's head set. When a user is busy, his/herportrait shows the back of his/her head. A preferred notificationchannel in this mode is a vibration through a pager unit. When a requesttimes out, callers may choose a different notification strategy, ifallowed. However, in this mode the contact's phone will never ring inpublic. Users can press a “Don't Answer” button to manually forestallnotifications by outside callers for a set time interval. This iscommunicated to callers by turning the contact's portrait into a graysilhouette. Offline communication is still possible in this mode,allowing the user to leave voicemail or a text message.

The above example demonstrates how the interruptiveness of notificationof a device such as a cell phone can be reduced by allowing a) thedevice to sense the attentive state of the user, b) the device tocommunicate this attentive state to subjects, and c) subjects to followsocial rules of engagement on the basis of this information. Secondly,interruptiveness is reduced by the device making intelligent decisionsabout its notification method on the basis of obtained information aboutthe user's attentive state.

Example 2 Telephone Proxy

Mediated communications systems such as a telephone typically requirecallers to interrupt remote individuals before engaging intoconversation. While previous research has focused on solving thisproblem by providing awareness cues about the other person'savailability for communication, there has been little work on supportingthe negotiation of availability that typically precedes communication inface-to-face situations. Face-to-face interactions provide a richselection of verbal and non-verbal cues that allow potentialinterlocutors to negotiate the availability of their attention withgreat subtlety.

In this example we present a mechanism for initiating mediatedconversations through eye contact. In our attentive telephone, referredto herein as “eyePHONE”, telephones were equipped with an attentive userinterface including an eye proxy and an eye contact sensor. The eyeproxy serves as a surrogate that indicates to a user the availabilityand attention of a remote user for communication, and the eye contactsensor conveys information about the user's attention to the remoteuser. Users initiate a call by jointly looking at each other's eyeproxy. This allows users to implement some of the basic social rules offace-to-face conversations in mediated conversations. This examplerelates to use of only two devices (telephones); however, it will beunderstood that this technology could be applied to any number ofdevices on a network.

The eye proxy consisted of a pair of Styrofoam® eyes, actuated by amotorized Sony EVI-D30 camera. The eyes were capable of rotating 180°horizontally and 80° vertically around their base. Eye contact of a userlooking at the eye proxy was detected by an eye contact sensor, asdescribed above (see FIG. 5), mounted above the eyes. Once the pupils ofa user were located, the proxy maintained eye contact by adjusting theorientation of the eyes such that pupils stayed centered within the eyecontact sensor image. Audio communication between eyePHONES wasestablished through a voice-over-IP connection.

To communicate the negotiation of mutual attention, we developed a setof gestures for eyePHONEs, shown in FIG. 6. With reference to FIG. 6,the following scenario illustrates how users may gradually negotiateconnections through these eye gestures: Connor wishes to place a call toAlex. He looks at Alex's proxy, which begins setting up a voiceconnection after a user-configurable threshold of 1.5 s of prolonged eyecontact. The proxy communicates that it is busy by iteratively glancingup—and looking back at Connor (see FIG. 6 b). On the other side of theline, Connor's proxy starts moving its eyes, and uses the eye contactsensor to find the pupils of Alex (see FIG. 6 a). Alex observes theactivity of Connor's proxy on his desk, and starts looking at theproxy's eye balls. When Connor's proxy detects eye contact with Alex,the eyePHONES establish a voice connection (see FIG. 6 c). If Alex doesnot want to take the call, he either ignores the proxy or looks awayafter having made brief eye contact. Alex's proxy on Connor's deskconveys Alex's unavailability by shaking its eyes, breaking eye contact,and not establishing a voice connection (see FIG. 6 d). If Connordecides his call is too urgent, he may choose to press a button thatproduces an audible ring. Optionally, calls may be set to completeautomatically when proxies determine a lack of eye contact over auser-configurable time period.

EyePHONES were also used to represent multiple participants duringconference calls. Unlike regular conference calls, the negotiation ofconnections using nonverbal cues allows group members to enter atdifferent times without interrupting the meeting. Furthermore, weimplemented a “cocktail party” feature to facilitate the establishmentof side conversations. When this is active, the speaker volume of aperson's proxy depends on the amount of eye contact received from thatperson.

Example 3 Audio/Video Applications

Attentive user interfaces using eye contact sensors may function todirect video cameras, or recording facilities, or to deliver audiovisualcontent. By mounting an eye contact sensor on a camera, and connectingits signal to the recording of this camera, an automated directionsystem can automatically switch to the camera currently looked at by apresenter.

Similarly, televisions and other audiovisual content delivery systemscan be augmented with eye contact sensors to determine whether thatcontent is being viewed, and to take appropriate action when it is nolonger viewed. In combination with a personal video recording system,this may involve tracking user attention automatically for variousshows, skipping commercials on the basis of perceived attentiveness,modulating volume level or messages delivered through that medium, orlive pausing of audiovisual material.

In a video conferencing system, eye contact sensors or related eyetracking technologies may be used to ensure that eye contact with a useris captured at all times, by switching to one of multiple cameraspositioned behind a virtual display such that the camera closest towhich the user is looking is always selected for broadcast. Quality ofservice of network connection, including resolution of audio and videodata can be modulated according to which person is being looked at, asmeasured by an eye contact sensor or other eye tracking device.

Example 4 Attention Monitor

As an attention monitor, an attentive user interface includes an eyecontact sensor, optionally in conjunction with other sensors formeasuring other indices of the attentive state of a user, and softwareto monitor what device, person, or task a user is attending to. Thisinformation can be used, for example, to determine the optimal channelof delivering information, prioritize the delivery and notification ofmessages, appointments, and information from multiple devices or usersacross a network, and generally manage the user's attention space.

As used herein, the term “attention space” refers to the limitedattention a user has available to process/respond to stimuli, given thatthe capacity of a user to process information simultaneously fromvarious sources is limited.

Software augmented with sensing systems including eye contact sensorsfunction as an intermediary to the management of a user's physicalattention. Thus, miniaturized eye contact sensors can be embedded in,and augment, small electronic devices such as PDAs, cell phones,personal entertainment systems, appliances, or any other object todeliver information when a user is paying attention to the device,deferring that information's delivery when the user's attention isdirected elsewhere. This information may be used, for example, todynamically route audio or video calls, instant messages, emailmessages, or any other communications to the correct location of theuser's current attention, and to infer and modulate quality of serviceof the network.

In environments with many potential subjects requesting a user'sattention, attentive user interfaces need a dynamic model of the user'sattentive context to establish a gradual and appropriate notificationprocess that does not overload the user. This context includes whichtask, device, or person the user is paying attention to, the importanceof that task, and the preferred communication channel to contact theuser. The invention provides a personalized communications server,referred to herein as “eyeREASON”, that negotiates all remoteinteractions between a user and attentive devices by keeping track ofthe user's attentive context. In one embodiment, eyeREASON is anadvanced personal unified messaging filter, not unlike an advanced spamfilter. EyeREASON decides, on the basis of information about the user'sprior, current, and/or future attentive state, the priority of a messageoriginating from a subject in relationship to that of tasks the user isattending to. By examining parameters of the message and user task(s),including attentive states of subjects pertaining to that message,eyeREASON makes decisions about whether, when, and how to forwardnotifications to the user, or to defer message delivery for laterretrieval by the user. A message can be in any format, such as email,instant messaging or voice connection, speech recognition, or messagesfrom sensors, asynchronous or synchronous. In the embodiment of speechrecognition and production interface, any speech communication between auser and device(s) can be routed through a wired or wireless headsetworn by the user, and processed by a speech recognition and productionsystem on the server. As the user works with various devices, eyeREASONswitches its vocabulary to the lexicon of the focus device, sendingcommands through that device's in/out (I/O) channels. Each devicereports to the eyeREASON server when it senses that a user is payingattention to it. EyeREASON uses this information to determine when andhow to relay messages from devices to the user. Using information aboutthe attentive state of the user, such as what devices the user iscurrently operating, what communication channels with the user arecurrently occupied, and the priority of the message relative to thetasks the user is engaged in, eyeREASON dynamically chooses an optimalnotification device with appropriate channels and levels ofnotification. Notifications can migrate between devices, tracking theattention of the user, as is illustrated by the below scenario. Oneapplication of eyeREASON is the management of prioritized delivery ofunified messages.

The following scenario illustrates interactions of a user with variousdevices enabled with attentive user interfaces, employing eye contactsensing capability, through eyeREASON's attentive reasoning system. Itshows how awareness of a user's attentive context may facilitateturn-taking between the user and remote ubiquitous devices. Alex entershis living room, which senses his presence (e.g., via the RF ID tag heis wearing) and reports his presence to his eyeREASON server. He turnson his television, which has live pausing capability (e.g., TiVo,personal video recorder (PVR)). The television is augmented with anattentive user interface having an eye contact sensor, which notifiesthe server that it is being watched. The eyeREASON server updates thevisual and auditory interruption levels of all people present in theliving room. Alex goes to the kitchen to get himself a cold drink fromhis attentive refrigerator, which is augmented with a RF ID tag reader.As he enters the kitchen, his interruption levels are adjustedappropriate to his interactions with devices in the kitchen. In theliving room, the TV pauses because its eye contact sensor reports thatno one is watching. Alex queries his attentive fridge and finds thatthere are no cold drinks within. He gets a bottle of soda from acupboard in the kitchen and puts it in the freezer compartment of thefridge. Informed by a RF ID tag on the bottle, the fridge estimates theamount of time it will take for the bottle to freeze and break. Itrecords Alex's tag and posts a notification with a timed priority levelto his eyeREASON server. Alex returns to the living room and looks atthe TV, which promptly resumes the program. When the notification timesout, Alex's eyeREASON server determines that the TV is an appropriatedevice to use for notifying Alex. It chooses the visual communicationchannel, because it is less disruptive than audio. A box with a messagefrom the fridge appears in the corner of the TV. As time progresses, thepriority of the notification increases, and the box grows in size on thescreen, demonstrating with increased urgency that Alex's drink isfreezing. Alex gets up, the TV pauses and he sits down at his computerto check his email. His eyeREASON server determines that the priority ofthe fridge notification is greater than that of his current email, andmoves the alert to his computer. Alex acknowledges this alert, andretrieves his drink, causing the fridge to withdraw the notification.Had Alex not acknowledged this alert, the eyeREASON server would haveforwarded the notification to Alex's email, or chosen an alternativechannel.

Example 5 Response Monitor

By placing an attentive user interface in the vicinity of any visualmaterial that one would be interested in tracking the response to, suchas advertisements (virtual or real), television screens, and billboards,the attention of users for the visual material can be monitored.Applications include, for example, gathering marketing information andmonitoring of the effectiveness of advertisements.

Example 6 Control of Graphical User Interface

An attentive user interface, using eye contact sensors or related eyetracking technology, can be used to modulate the amount of screen spaceallocated to a window in a graphical user interface windowing systemaccording to the amount of visual attention received by that window.Similarly, attentive user interfaces employing eye contact sensors orother related eye tracking technology may be used to initiate theretrieval of information on the basis of progressive disclosure. Forexample, information may initially be shown with limited resolution onthe side of a display. When a user looks at the representation for a setamount of time, more detailed information is retrieved and rendered onthe screen using a larger surface. Examples include stock market tickersthat grow and provide more information when users pay attention to it,instant messaging buddy status lists that engage in connections, openingup chat boxes with users that are being looked at, etc.

Example 7 Graphical User Interface

This example relates to use of an attentive user interface in awindowing system, referred to herein as “eyeWINDOWS”, for a graphicaluser interface which incorporates fisheye windows or views that use eyefixation, rather than manual pointing, to select the focus window. Thewindowing system allocates display space to a given window based on theamount of visual attention received by that window. Use of eye inputfacilitates contextual activity while maintaining user focus. It allowsmore continuous accommodation of the windowing system to shifts in userattention, and more efficient use of manual input.

Windowing systems of commercial desktop interfaces have experiencedlittle change over the last 20 years. Current systems employ the samebasic technique of allocating display space using manually arranged,overlapping windows into the task world. However, due to interruption byfor example, system prompts, incoming email messages, and othernotifications, a user's attention shifts almost continuously betweentasks. Such behavior requires a more flexible windowing systems thatallows a user to more easily move between alternate activities. Thisproblem has prompted new research into windowing systems that allow morefluent interaction through, e.g., zooming task bars (Cadiz et al., 2002)or fisheye views (Gutwin, 2002). While most of this work emphasizes theuse of manual input for optimizing display space, there has been littlework on windowing systems that sense the user's attention using moredirect means. Using an alternate channel for sensing the attention ofthe user for parts of a display has a number of benefits. Firstly, itallows an undisrupted use of manual tools for task-oriented activities;and secondly, it allows a more continuous accommodation of shifts inuser attention.

Consider, for example, a scenario where a user is working on a task on apersonal computer when an alert window appears on the screen to informhim that a new email message has just been received. The alert windowobscures the user's current task and the received message, such that theuser is only allowed to resume his task or read the message aftermanually dismissing the alert. Tracking the focus of a user allows aninterface to more actively avoid interrupting the user, e.g., by morecareful placement of windows.

Use of eye input to select a window of interest has several advantages.Firstly, the eyes typically acquire a target well before manual pointingis initiated (Zhai, 2003). Secondly, eye muscles operate much fasterthan hand muscles (Zhai, 2003). Finally, the eyes provide a morecontinuous signal that frees the hands for other tasks. Bolt (1985)recognized early on how, using a pair of eye tracking glasses, windowsmight automatically be selected and zoomed.

Unfortunately, his glasses did not provide sufficient resolution.However, recent advances allow seamless integration of an eye trackerwith a head movement tolerance of 60 cm and an on-screen accuracy ofbetter than 1 cm into a 17″ LCD screen. We used a similar eye tracker toimplement eyeWINDOWS.

To determine which window should be the focus window, eyeWINDOWSobserves user eye fixations at windows with an LC Technologies eyetracker. Using a lens algorithm similar to Sarkar et al. (1992), thefocus window is zoomed to maximum magnification. Surrounding windowscontract with distance to the focus window. However, the enlarged windowdoes not obscure the surrounding contracted windows, such that the usercan readily view all windows. While typical fisheye browsers run withina single window, eyeWINDOWS affects all active applications. Traditionalicons are replaced with active thumbnail views that provide fullfunctionality, referred to herein as “eyecons”. Eyecons zoom into afocus window when a user looks at them.

Our first design issue was that of when to zoom an eyecon into a focuswindow. We first experimented with a continuous fisheye lens, whichshifted whenever the user produced an eye movement. This led to focustargeting problems similar to those observed during manual pointing(Gutwin, 2002). In subsequent implementations, the lens was shifted onlyafter selecting a new focus window. Our second design issue was how totrigger this selection. We designed two solutions. In our firstapproach, dwell time was used as a trigger. An eyecon zooms into a focuswindow after a user-configurable period of fixations at that eyecon. Toavoid a Midas Touch effect (Zhai, 2003)—where users avoid looking toprevent unintentional triggering—fisheye magnification is applied withnon-linear acceleration. When the user first fixates on an eyecon, itstarts growing very slowly. If this is not what the user intended, onefixation at the original focus window undoes the action. However, whenthe user continues to produce fixations at the eyecon, zoomingaccelerates until maximum magnification is reached. Our second approachto this problem prevents a Midas Touch effect altogether. In thisapproach, a new focus window is selected when the user presses the spacebar while fixating at an eyecon. Focus window selection is suspendedduring normal keyboard or pointing activity, such as when scrolling ortyping. Fish-eye magnification does not apply to certain utilitywindows, such as tool bars.

Initial user observations appear to favor the use of key triggering forfocus window selection. The following scenario illustrates this process:a user is working on a text in the focus window in the center of thescreen. The focus window is surrounded by eyecons of related documents,with associated file names. The user wishes to copy a picture from thedocument to the right of his focus window. He looks at its eyecon andpresses the space bar, and the eyecon zooms into a focus window, whilethe old focus window shrinks into an eyecon. After having found thepicture, he places it in the clipboard and shifts his attention back tothe original document. It zooms into a focus window and the user pastesthe picture into the document. This scenario illustrates how contextualactions are supported without the need for multiple pointing gestures toresize or reposition windows. EyeWINDOWS also supports moreattention-sensitive notification. For example, the user is notified of amessage by a notification eyecon at the bottom of the screen. When theuser fixates at the notification eyecon it zooms to reveal its message.The notification is dismissed once eyeWINDOWS detects the message wasread. This illustrates how an attentive user interface supports userfocus within the context of more peripheral events.

Example 8 Attentive Appliance

Any household or commercial/industrial appliance, digital or analogapparatus, or object may be configured as an attentive appliance. Suchattentive appliance may be a stand alone “smart appliance”, or may benetworked to a shared computational resource such as a communicationsserver (e.g., eyeREASON; see Example 4), providing unified messagecapabilities to all networked appliances without requiring extensiveembedded computational support in each appliance. In Example 4 theattentive refrigerator was a refrigerator augmented with thecapabilities to sense eye contact with its user, presence of objectsinside and outside the fridge through radio frequency ID tags, useridentification and presence sensing through RF ID tags or any othermeans of sensing, as well as identification of objects inside andoutside the fridge. A small computer embedded in the fridge, andconnected to a network through a tcp ip connection, runs a simpleprogram that allows the fridge to reason about its contents, andinteract with the user, by incorporating eye contact with the user. Thefridge may contain software for processing and producing speech, and aspeech recognition and production engine residing on eyeREASON canadvantageously be employed to process speech for it, responding tocontextualized verbal queries by a user. This is accomplished by sendingxml speech recognition grammars and lexicons from the fridge toeyeREASON that are contextualized upon the state of the fridge's sensingsystems. The fridge will send xml grammars and enable speech processingwhenever a user is in close proximity to it, and/or making eye contactwith the fridge, and/or holding objects from the fridge in his/her hand.The user is connected to the speech recognition and production engine oneyeREASON through a wireless headset (e.g., BlueTooth®). This allowseyeREASON to process speech by the user, with the contextualizedgrammars provided by the appliance the user is interacting with.EyeREASON determines a) whether speech should be processed; e.g., focusevents sent by the appliance on the basis of information from its eyecontact sensor; b) for which appliance, and with which grammar speechshould be processed; c) what commands should be sent to the appliance asa consequence; and d) what the priority of messages returned from theappliance should be. Messages sent by appliances during synchronousinteractions with a user will receive the highest notification levels.

The following scenario illustrates the process: User A is standing nearhis attentive fridge. He asks what is contained in the fridge whilelooking at the fridge. The fridge senses his presence, detects eyecontact, and determines the identity of the user. It sends an xmlgrammar containing the speech vocabulary suitable for answering queriesto user A's eyeREASON server. The eyeREASON server switches its speechrecognition lexicon to process speech for the fridge, as instructed bythe current xml grammar. It parses the user's speech according to thegrammar, recognizes that the user wants a list of items in the fridge,and sends a command to the fridge to provide a list of items, accordingthe xml specs. The fridge responds by sending a text message toeyeREASON listing the items in the fridge. Since the user is directlyengaged in a synchronous interaction with the fridge, eyeREASON decidesthe message should be forwarded to the user immediately. Since the userhas been interacting with the fridge through speech over his headset,eyeREASON uses this same path, speaking the message to the user with itsspeech production system. The user opens the fridge and retrieves somecheese. The fridge recognizes that the hand of user A is in the fridge,and has removed the cheese. It sends a hand focus event, andsubsequently an object focus event to the eyeREASON server with the RFID of the cheese object, with corresponding grammar for handling anyuser speech. The user may query any property of the cheese object, forexample its expiration date. If the user says “start message” eyeREASONwill record any voice message and tag it with the RF ID of the objectthe user was holding, as well as the ID of the user. It will stoprecording when user puts the object back into the fridge, tagging theobject with a voice message. It forwards this voice message with a storecommand to the embedded processor in the fridge. The next time any userother than user A retrieves the same object, the fridge will forward thevoice message to pertaining to this object to that user.

Any attentive appliance may signal its attention for a user using, forexample, an eye proxy mounted in close proximity to it. The eye proxy(described in more detail above and in Example 2) will function in lieuof an eye contact sensor, tracking and maintaining eye contact with auser. It maintains activation of the speech recognition engine for theappliance it is associated with while there is sufficient statisticalevidence the user is looking at or interacting with that appliance.Before replying to a user through a message, the appliance will attemptto signal its request for attention by seeking eye contact between itsproxy and user. Should the user not respond, the eyeREASON system willdetermine a new notification level for the message. EyeREASON will lowerthe notification level of the message the moment a user is perceived tobe no longer interacting directly with the appliance that sent themessage. Competing with other messages in the priority queue of theuser, the server will either forward the message, for example to theuser's cell phone, or store it for later retrieval in the user's messagequeue. If the priority of the message is determined higher than those ofother messages in the user's notification queue, eyeREASON will attemptto progressively notify the user of the message up to a user determinednumber of times. Each time the user does not respond the notificationlevel of the message is increased. This allows eyeREASON to seekdifferent channels of notification each time the notification isre-triggered. For example, it may initially attempt to signal attentionthrough seeking eye contact with the user through the eye proxypertaining to the appliance that sent the message. When this fails, itmay initiate a low-volume auditory interruption in that appliance. Whenthis fails, it may forward the notification to the appliance the user iscurrently interacting with, potentially disrupting the user's currentactivity. The latter should only occur when messages are determined tobe of a greater notification level than the user's current tasks. Whenthis fails, the message is forwarded to the user's message queue forlater retrieval.

Those of ordinary skill in the art will recognize, or be able toascertain through routine experimentation, equivalents to theembodiments described herein. Such equivalents are within the scope ofthe invention and are covered by the appended claims.

REFERENCES

-   Bolt, R. A., 1985. Conversing with Computers. Technology Review    88(2), pp. 34-43.-   Cadiz, J. et al. (2002). Designing and Deploying an Information    Awareness Interface. In: Proceedings of CSCW'02.-   Frolich, D., et al., 1994. Informal Workplace Communication: What is    It Like and How Might We Support It? HP Tech. Report.-   Gutwin, C., 2002. Improving Focus Targeting in Interactive Fisheye    Views. In: Proceedings of CHI'02, pp. 267-274.-   Morimoto, C. et al., 2000. Pupil Detection and Tracking Using    Multiple Light Sources. Image and Vision Computing, vol 18.-   Sarkar, M. et al., 1992. Graphical Fisheye Views of Graphs. In:    Proceedings of CHI'92, pp. 83-91-   Vertegaal, R., 1999. The GAZE Groupware System. In Proceedings of    CHI'99. Pittsburgh: ACM.-   Zhai, S., 2003. What's in the Eyes for Attentive Input. In:    Communications of ACM 46(3).

1-44. (canceled)
 45. Apparatus for communication between a user and adevice, comprising: a hardware sensor in or on the device that sensesattention of the user specifically toward the device; and a processorthat processes a signal from the hardware sensor and outputs to thedevice a measure or index of the user's attention toward the device; andwherein operation of the device is modulated on the basis of the measureor index of the user's attention toward the device; wherein theoperation that is modulated is initiated by the device and provides anotification and/or information and/or communication to the user basedon the user's attention toward the device.
 46. The apparatus of claim45, wherein the hardware sensor senses attention of the user by sensingone or more of eye contact, eye movement, eye position, eye gazedirection, voice, body presence, body orientation, head and/or faceorientation, user activity, and brain activity/arousal.
 47. Theapparatus of claim 45, wherein the hardware sensor senses attention ofthe user by sensing one or more of eye contact, eye gaze direction, bodyorientation, head and/or face orientation, and user activity.
 48. Theapparatus of claim 46, wherein the hardware sensor senses eye gazedirection of the user.
 49. The apparatus of claim 46, wherein thehardware sensor senses eye contact of the user.
 50. The apparatus ofclaim 49, wherein the hardware sensor: obtains successive full-framevideo fields of alternating bright and dark video images of the pupils;and subtracts the images between frames to locate the pupils.
 51. Theapparatus of claim 50, wherein the hardware sensor: detects a glint inthe eyes; and confirms eye contact when the glint is aligned with thepupils.
 52. The apparatus of claim 45, wherein, when the user'sattention is not directed toward the device, modulating comprises thedevice notifying the user progressively, from a less interruptivenotification to a more interruptive notification.
 53. The apparatus ofclaim 52, wherein the notification is of at least one type selected fromthe group consisting of auditory, visual, and tactile.
 54. The apparatusof claim 45, wherein the hardware sensor is attached to or embedded inthe device.
 55. The apparatus of claim 45, further comprising thedevice.
 56. The apparatus of claim 45, wherein the device is selectedfrom an electronic device, a machine, and furniture.
 57. The apparatusof claim 45, wherein the device is a personal computer.
 58. Theapparatus of claim 45, wherein the device is a cellular telephone. 59.The apparatus of claim 45, wherein the device is a telephone.
 60. Theapparatus of claim 45, wherein the device is a personal digitalassistant (PDA).
 61. The apparatus of claim 45, wherein the device is anappliance.
 62. The apparatus of claim 45, wherein operation of thedevice is modulated by forwarding a notification from the device toanother device or a network of devices.
 63. The apparatus of claim 45,wherein operation of the device is modulated by modulating anotification being sent to the user by the device.
 64. The apparatus ofclaim 45, wherein operation of the device is modulated by forwarding themeasure or index of the user's attention to another device or a networkof devices.
 65. The apparatus of claim 45, wherein the operation that ismodulated is selected from notification, communication, informationtransfer, and a combination thereof.
 66. Apparatus for communicationbetween a user and a network of devices, comprising: a hardware sensorin or on each device of the network of devices, each hardware sensorsensing attention of a user specifically toward its respective device;and a processor that processes signals from the hardware sensors andoutputs a measure or index of the user's attention toward the devices;wherein operation of one or more of the devices is modulated on thebasis of the measure or index of the user's attention toward thedevices; wherein the operation that is modulated is initiated by atleast one of the devices and provides a notification and/or informationand/or communication to the user based on the user's attention towardthe devices.
 67. The apparatus of claim 66, wherein the operation thatis modulated is selected from notification, communication, informationtransfer, and a combination thereof.
 68. The apparatus of claim 66,wherein modulating comprises routing the notification, communication,information transfer, or combination thereof, to a device to which theuser's attention is directed.
 69. The apparatus of claim 66, whereinmodulating further comprises modulating notification of the userprogressively, from a less interruptive notification to a moreinterruptive notification.
 70. The apparatus of claim 66, whereinsensing the user's attention comprises sensing eye contact of the userwith each device.
 71. The apparatus of claim 66, wherein each device isindependently a personal computer, cellular telephone, a telephone, apersonal digital assistant (PDA), or an appliance.
 72. The apparatus ofclaim 66, further comprising one or more of the devices.
 73. Apparatusfor communication between users and devices, comprising: a firsthardware sensor in or on a first device of a network of devices, thefirst hardware sensor sensing a first user's attention specificallytoward the first device; a second hardware sensor in or on a seconddevice of a network of devices, the second hardware sensor sensing asecond user's attention specifically toward the second device; a firstproxy associated with the first device, the first proxy communicating tothe first user the second user's attention toward the second device; asecond proxy associated with the second device, the second proxycommunicating to the second user the first user's attention toward thefirst device; a processor that processes signals from the first andsecond hardware sensors and relays to the network a measure or index ofthe first and second users' attention toward the respective first andsecond devices; wherein communication between the first and seconddevices is modulated on the basis of the measure or index of the firstand second users' attention toward their respective devices and providesa notification to the first and second users.
 74. The apparatus of claim73, wherein communication between the first and second devices isenabled when respective proxies indicate that respective attention ofthe first and second users are toward their respective devices.
 75. Theapparatus of claim 73, wherein the network comprises more than twodevices.
 76. The apparatus of claim 73, wherein each device isindependently a personal computer, cellular telephone, a telephone, apersonal digital assistant (PDA), or an appliance.
 77. The apparatus ofclaim 73, further comprising one or more of the devices.